Papers by Richard Yuanzhe Pang
Towards Actual (Not Operational) Textual Style Transfer Auto-Evaluation (D19-55)
Copied to clipboard
| Challenge: | elucidates the dangerous current state of style transfer auto-evaluation research. |
| Approach: | They propose ways to aggregate the three metrics into one evaluator. |
| Outcome: | The proposed method could be used to aggregate the three metrics into one evaluator. |
Unsupervised Evaluation Metrics and Learning Criteria for Non-Parallel Textual Transfer (D19-56)
Copied to clipboard
| Challenge: | Existing methods for textual transfer with no parallel corpora are insufficient to evaluate textual paraphrases with modified attributes or properties. |
| Approach: | They propose to add a metric for post-transfer classification accuracy and a method to combine them into a single overall score. |
| Outcome: | The proposed metrics correlate well with human judgments, at both the sentence-level and system-level. |
Token Dropping for Efficient BERT Pretraining (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to accelerate pretraining of transformer-based models are computationally expensive and degrade performance on downstream tasks. |
| Approach: | They propose a "token dropping" method to accelerate the pretraining of transformer-based models by 25% . they leverage the already built-in masked language modeling loss to identify unimportant tokens with practically no computational overhead. |
| Outcome: | The proposed method reduces the pretraining cost of BERT models by 25% while achieving similar overall performance on downstream tasks. |
Comparing Test Sets with Item Response Theory (2021.acl-long)
Copied to clipboard
Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman
| Challenge: | Recent results from large pretrained models show that many datasets are saturated and unlikely to detect further progress. |
| Approach: | They evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. |
| Outcome: | The proposed datasets are saturated and unlikely to detect future improvements. |
ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | a non-autoregressive machine translation model can minimize the autoregressive teacher's energy . engINE is an inference network trained to minimize the teacher' energy based on distilled corpora . |
| Approach: | They propose to train a non-autoregressive machine translation model to minimize autoregressive teacher energy by using an inference network instead of distilled corpora. |
| Outcome: | The proposed model achieves state-of-the-art non-autoregressive results on two datasets . the proposed model is trained to minimize the autoregressive teacher energy . |
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020.acl-main)
Copied to clipboard
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman
| Challenge: | Unsupervised pretraining has recently pushed the state of the art on many natural language understanding tasks. |
| Approach: | They perform a large-scale survey on a pretrained RoBERTa model with 110 intermediate-target task combinations and 25 probing tasks to reveal the specific skills that drive transfer. |
| Outcome: | The proposed model is trained on 110 intermediate-target task combinations and compared with 25 probing tasks to reveal the specific skills that drive transfer. |
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2023.acl-long)
Copied to clipboard
Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang, Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang, Jason Phang, Samuel R. Bowman
| Challenge: | Getting sociological beliefs wrong can slow research and lead to wasted effort, missed opportunities, and needless fights. |
| Approach: | They present the results of the NLP Community Metasurvey, run from May to June 2022. |
| Outcome: | The NLP community metasurvey elicited opinions on controversial issues from May to June 2022. |
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization datasets often have issues that seriously limit their usability. |
| Approach: | They propose a faster but more straightforward approach to developing summarization benchmark data . they use a protocol that hires highly-qualified contractors to read stories and write original summaries from scratch . |
| Outcome: | The proposed protocol is faster but more straightforward than scraping summaries from everyday text. |
Self-Generated Critiques Boost Reward Modeling for Language Models (2025.naacl-long)
Copied to clipboard
Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, Rui Hou
| Challenge: | Existing reward models produce scalar scores and struggle to incorporate critiques in a natural language format. |
| Approach: | They propose a framework that predicts critiques and rewards using self-generated critiques without extra supervision. |
| Outcome: | The proposed framework improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges. |
QuALITY: Question Answering with Long Input Texts, Yes! (2022.naacl-main)
Copied to clipboard
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, Samuel Bowman
| Challenge: | Existing models for natural language understanding are limited to processing only a few hundred words at a time. |
| Approach: | They propose a dataset with context passages in English that have an average length of 5,000 tokens. |
| Outcome: | a new dataset with long-text comprehension questions is used to test models on long-document comprehension . the questions are validated by contributors who have read the entire passage, not just excerpts . only half of the questions can be answered by annotators working under tight time constraints . |
Consistency of a Recurrent Language Model With Respect to Incomplete Decoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural sequence models trained with maximum likelihood have been shown to exhibit issues such as length bias and degenerate repetition. |
| Approach: | They propose to use a recurrent language model to address inconsistency in decoding algorithms that are inconsistent despite the fact that recursive language models are trained to produce sequences of finite length. |
| Outcome: | The proposed methods prevent inconsistency in the proposed models. |
Leveraging Implicit Feedback from Deployment Data in Dialogue (2024.eacl-short)
Copied to clipboard
| Challenge: | Xu et al., 2023) and Bai ed., 2019) use crowdworkers to collect signals from natural dialogue episodes. |
| Approach: | They use the publicly released BlenderBot deployment data to extract signals from conversations to implicitly measure the quality of a machine-generated utterance. |
| Outcome: | The proposed model improves over baseline models, but some proxy signals can lead to undesirable generations. |
AgreeSum: Agreement-Oriented Multi-Document Summarization (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on agreement-oriented multidocument summarization have focused on clusters of articles . a recent study focused on the use of a pretraining framework to summarize articles based on the "union" of the articles. |
| Approach: | They propose to use agreement-oriented multidocument summarization to provide agreement-orientated summaries that represent information common to all articles. |
| Outcome: | The proposed task is called agreement-oriented multidocument summarization . the authors apply the pretrained model PEGASUS onto the task . |
AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following (2026.acl-long)
Copied to clipboard
Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nanshu Wang, Xiaoliang Peng, Beibin Li, Shengjie Bi, Shishir G Patil, Qi Qi, Shengyu Feng, Julian Katz-Samuels, Richard Yuanzhe Pang, Sujan Kumar Gonugondla, Hunter Lang, Yue Yu, Yundi Qian, Maryam Fazel-Zarandi, Licheng Yu, Amine Benhalloum, Hany Hassan Awadalla, Manaal Faruqui
| Challenge: | Recent advances in large language models (LLMs) have shown impressive performance on a range of tasks, yet advanced instruction following (IF) remains a significant challenge. |
| Approach: | They propose a benchmark that features over 1,600 prompts and expert-curated rubrics that assess LLMs’ ability to follow complex, multi-turn, and system-level instructions. |
| Outcome: | The proposed framework improves instruction-following abilities of large language models, achieving a 6.7% gain on AdvancedIF and strong results on public benchmarks. |
Reward Gaming in Conditional Text Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work has used reward functions learned from human annotations to align conditional text generation models with desired behaviors. |
| Approach: | They propose to use reinforcement learning to train conditional text generation models with reward functions learned from human annotations to align outputs with desired behaviors. |
| Outcome: | The proposed framework improves the quality of generated summaries by using saliency and faithfulness metrics. |